Design of Simulator Scenarios to Study the Effectiveness of Electronic Stability Control Systems

The mission of the National Advanced Driving Simulator is to conduct highway safety research that will reduce the annual loss of life on US roadways. The NADS is particularly well suited in its ability to realistically replicate vehicle dynamics, and associated motion and visual cues, in order to conduct complex experiments. It is unique in its ability to study vehicle control and loss of control situations in a safe and controlled environment. These capabilities make it a perfect device to study the effectiveness of electronic stability control systems (ESC) where proper handling during loss of vehicle control is critical to assess system effectiveness. This paper focuses on the challenges associated with creating repeatable, yet unexpected, scenario events where loss of control is imminent for most drivers. The paper details the scenario events designed for a large-scale experiment to study ESC, discusses data derived from these scenarios, and presents findings of scenario effectiveness. The paper provides a discussion of what constitutes loss of control and how to effectively measure its effect. INTRODUCTION The mission of the National Advanced Driving Simulator (NADS) is to conduct highway safety research that will reduce the annual loss of approximately 42,000 lives due to vehicle collisions (1). The NADS is unique in its ability to facilitate research in a highly realistic, controlled, and repeatable environment that would be too dangerous or expensive to perform on the real road. The NADS is particularly effective in modeling complex vehicle dynamics and subsystems while providing appropriate cues to the driver. These cues facilitate realistic responses from the roadway, vehicle and driver interaction. The research performed on the NADS can be used to assess the value of various vehicle systems, to study the effects of prescription and illicit substances on driving, and to formulate new legislation that will make roads safer for everyone. This paper focuses on the challenges associated with creating repeatable yet unexpected scenarios for a study assessing the effectiveness of electronic stability control systems (ESC) [4]. This paper describes the scenarios used and presents data on the adequacy of these scenarios in assessing ESC effectiveness. MOTIVATION ESC is a system designed to alleviate crashes where the root cause is loss of control. Statistical studies have shown that ESC can reduce accidents [3, 5, 6, 7, 8]; however, there has been no empirical research on ESC effectiveness until recently, when a first-ofits-kind assessment of ESC was performed at NADS [10]. This is a second study whose goal was to further study ESC under realistic conditions in various scenarios where loss of control was likely, contributed in part by wet pavement conditions. A secondary goal was to study participants’ reactions, understanding, and monitoring of ESC. ESC effectiveness was measured by comparing the outcome of scenarios with and without the ESC system. In the remainder of this paper, the term ON is used to indicate that the ESC system is present and activates when necessary to prevent loss of control. The term OFF is used to indicate that the ESC system is disabled and will not activate under any circumstances. There are many challenges to designing simulator scenarios that are realistic, unexpected, and fitting for the research goals. There are especially unique challenges in designing scenarios for studies involving loss of control because the driver can easily become hyper-vigilant or nonrespondent after repeated exposure to too many intense situations. ESC is only useful in loss of Watson, Papelis and Ahmad 3 control situations that typically occur during unexpected events that require hard steering inputs. These scenarios tend to be memorable to drivers, thus the scenarios used for a study must be diverse but comparable. The solution posed in this study was to implement scenarios tha t required last-second steering to avoid a vehicle incurring from the left lane, the right lane, and head-on, along with events causing the driver to respond to unforeseen wind gusts and a difficult geometric challenge posed in the form of a closing radius turn. The events proved to be unique and unanticipated by drivers. Each of the events is described in the following section. ESC SCENARIOS This study utilized five scenarios to study how the driver responds to different situations with and without the help of the ESC system. An ESC equipped model of an Oldsmobile Intrigue was used [9]. All of the scenarios utilized a visual database that was built specifically for this study. Figure 1 shows a top-down view of the entire visual database used in this study, along with locations of the five scenario events. The database consisted mostly of twolane rural highways along with some fourlane highways. All roads were built with shoulders that featured traction, vibration, and audio characteristics that differed from the on-road pavement. This more realistically simulated the various moments and the slippage that occurs when some of a vehicle’s tires depart the roadway. Although the scenarios occurred in different parts of the database, the entire virtual world was designed to look similar. All scenarios presented scenery that was typically associated with a rural two-lane highway surrounded by farms and rolling hills. One of the scenario events involved a wind gust. In addition to the dynamic simulation of the wind effects on the vehicle, the visual database includes special effects (smoke stacks) that suggested the existence of strong wind. FIGURE 1 A top-down view of the entire visual database Watson, Papelis and Ahmad 4 All scenarios started on two-lane highways that had a posted speed limit of 65 mph. All scenarios consisted of the same type of vehicles and traffic, with some vehicles parked on the shoulders. The similarity in general scenario features was planned so the driver would get accustomed to the various types of vehicles and not anticipate the likelihood of an event based on scenery or certain vehicle types. The vehicles that performed avoidance functions in particular events had been observed before by the driver but without the need for a response by the driver. Each scenario event was run separately, and the order was systematically varied so that the effects of scenario order were equally balanced across all drivers in the experiment. Each of the five scenarios started in almost exactly the same way. The driver’s vehicle was parked on the shoulder of a rural twolane highway. The driver was instructed to wait until a red vehicle drove by, and then instructed to pull off the shoulder onto the road and begin driving. The initial posted speed limit was 65 mph for all scenarios. The red vehicle served as the lead vehicle for all scenarios. All scenarios were driven on wet pavement. The virtual environment reflected conditions consistent with wet pavement. In particular, the scene was cloudy and the pavement appeared wet by using object reflections on the road. In each scenario, the driver had the opportunity to earn incentive pay by keeping their velocity between 62 and 68 mph. Each scenario was designed to be a few minutes in length. The following subsections describe each of the five ESC scenarios in detail. Decreasing Radius Curve Scenario This scenario was designed to expose the driver to a situation requiring difficult maneuvering of the vehicle around a sharp curve. The curve was designed to be marginally drivable at the posted speed limit. The driver began the scenario on a rural two-lane highway with a 65 mph posted speed limit. A lead vehicle passed, and then the driver pulled off the shoulder onto the road and began to drive. The lead vehicle remained within view. During the short drive between the start and the curve, there were several vehicles that drove by the driver in the oncoming lane. The oncoming vehicles consisted of several types of vehicles, including those vehicles that served specific purposes in other scenarios. There were also several parked vehicles on the opposite shoulder. The purpose of these vehicles, other than contribute to overall scene complexity, was to allow the participant to become accustomed to their existence, so that when such vehicles were part of an event in other scenarios, there would be no predisposition to associate any emergency meaning to their existence. After a few minutes, the driver and lead vehicle approached a turn, as shown in Figure 2. A 50 mph speed limit warning sign was located before the right turn. The lead vehicle negotiated the first curve at high speed with no problems. The driver should have also negotiated the initial curve with no problem. After the first curve, the driver encountered a 45 mph speed limit warning sign before the second curve. The lead vehicle negotiated the second curve, with the sharp decreasing radius of curvature, without any hesitation. The driver then encountered the same sharp curve. The decreasing radius of curvature was not apparent until after the driver was well into the curve. Oncoming vehicles kept the driver from cutting the curve. Watson, Papelis and Ahmad 5 FIGURE 2 A top-down view of decreasing radius curve scenario. All radius of curvature figures are measured to the road centerline . Right Incursion Scenario The goal of this scenario was to force the driver to perform a double lane change at high speed. A vehicle pulling out of a hidden driveway attached to a roadside farm house, combined with carefully timed oncoming traffic, created the conditions for such a maneuver. The scenario began the same way as the previous scenario, with the driver parked on the shoulder of a rural twolane highway with a posted speed limit of 65 mph. A vehicle passed the driver and the driver pulled off the shoulder onto to the road and began driving. Along the way, the driver encountered vehicles in the oncoming lane and parked vehicles on the side of the road. In addition, the driver encountered several farm houses, located on both sides of the highway, with driveways similar to the one used in the incursion event. The actual incursion event started when the driver approached the specific driveway with the hidden right incursion vehicle, as shown in Figure 3. When the driver was 1.8 seconds from arriving at the driveway location, the hidden parked vehicle pulled out from the right and into the driver’s lane. The time-to-collision (TTC) was such that braking alone would not avoid the collision. A vehicle in the oncoming lane arrived at the incursion point approximately 2 seconds after the driver. The spacing was such that the driver had enough time to perform the double lane change without colliding with the oncoming vehicle. Parked vehicles on the left shoulder prevented the driver from avoiding the oncoming traffic by going to its left. Wind Gust Scenario The goal of this scenario was to create a temporary but significant high-speed steering instability in order to assess the driver’s ability to recover. Strong wind gusts from the right were implemented in this event. Figure 4 shows the profile of the wind intensity plotted over distance traveled. Given the lack of a validated aerodynamic model, there was no easy way to determine the actual velocity of the wind gusts. Therefore, their magnitude was experimentally adjusted so that if no steering correction was made, the participant’s vehicle would drift into the center of the opposite lane. Watson, Papelis and Ahmad 6 FIGURE 3 The right incursion scenario FIGURE 4 Wind gust magnitudes Watson, Papelis and Ahmad 7 Just like the other scenarios, the driver began on a two-lane rural highway with a posted speed limit of 65 mph. Along the way, the drive r encountered oncoming traffic and parked traffic on the side of the road. In addition, various smoke stacks suggested the presence of gusty winds. The actual event started as a series of wind gusts from the right that pushed the driver to the left, toward oncoming traffic, as shown in Figure 5. The wind came and went in varying amplitudes for approximately 5 seconds. As the driver responded to the wind, the wind stopped, so the driver had to correct the steering offset in the other direction. FIGURE 5 The wind gust scenario Left Incursion Scenario The goal of this scenario was to force the driver to react to an incursion from the left and apply a significant steering input while traveling at high speed. Oncoming traffic veered into the driver’s lane. The driver started out driving on a two-lane rural highway with a posted speed limit of 65 mph. Along the way, the driver encountered oncoming traffic and several parked traffic on the shoulders. As the driver approached the location of the event, as shown in Figure 6, one of the oncoming vehicles was tasked to arrive at the event location at a fixed relative position to the driver. As the driver approached the event area, oncoming traffic approached a parked vehicle blocking the shoulder opposite to the driver’s lane. The blocking vehicle began moving onto the road from the shoulder and cut off the oncoming traffic that drifted into the driver’s lane to avoid a collision. The driver was forced to respond by veering to the right to avoid a collision. Watson, Papelis and Ahmad 8 FIGURE 6 The left incursion scenario Obstacle Avoidance Scenario The goal of this scenario was to expose the driver to an unexpected head-on obstacle that required significant control input for avoidance. The event was motivated by a van dropping cargo onto the road in front of the driver. The driver pulled onto a two-lane rural road just like in the other scenarios. After a few minutes of driving, the two-lane highway transitioned into a fourlane highway. A few vehicles passed the driver on the left lane and changed into the right lane in front of the driver. The purpose of this was to make the driver accustomed to traffic overtaking the driver’s vehicle. With the driver in the right lane, the event began with a van driving past the driver in the left lane. The van moves 1.4 seconds ahead of the driver and then performed a right lane change to move into to the driver’s lane ahead of the driver.. The van maintained a headway of 1.4 seconds from the driver for approximately 30 seconds. The van drove over a bump and its rear doors swung open, dropping a desk onto the road in front of the driver, as shown in Figure 7. The driver was forced to veer left or right to avoid hitting the desk. DISCUSSION OF CHALLENGES One of the challenges in assessing the effectiveness of the scenarios, as well as the ESC system, was the determination of loss of control (LOC). Part of the challenge was that LOC is, to some degree, a subjective measure. If judged purely on a mathematical basis, some amount of LOC takes place every time the vehicle turns, as the tires slip laterally in order to create turning forces. The driver’s intent and experience also complicated judgment of LOC. Experienced drivers, such as those involved in competitive racing, routinely operate the ir vehicles in a manner that a normal driver would consider out of control. An added practical complication was the fact that the simulator, in order to prevent an abrupt stop after high-G accelerations induced during severe maneuvers, often terminated the simulation before the outcome of the scenario had time to fully unravel. Typically, this happened when yaw rates had built up to excessive amounts that Watson, Papelis and Ahmad 9 precluded recovery, even with the help of ESC; however, in some cases the yaw rate was low enough that recovery would have been possible. FIGURE 7 Obstacle avoidance All of the factors listed above necessitated the development of an algorithm to detect LOC. Participants in this study were ordinary drivers, so the assessment of LOC had to focus on a compromise between a mathematical and subjective definition. The algorithm utilized a definition of LOC that depended on excessive slip rates and terminal vehicle orientation. To better assess scenario and ESC effectiveness, a second independent performance measure was also calculated. Road Departure (RD) is a measure indicating whether the driver came to rest beyond the shoulder boundary. The reason for incorporating RD in the performance measures was to capture the ability of ESC to maintain directional control, even when a low coefficient of friction allows the vehicle to skid beyond the road boundary. In effect, the worst outcome is LOC; in this case the driver has little or no control over the direction that the vehicle is facing and yaw rates build up to the point where the vehicle may spin beyond the point of recovery. An RD is “better” than LOC in terms of likelihood of severe accident or injury because the vehicle’s orientation is still under control, but the actual path traveled is not. The reason that this outcome is considered less severe than LOC is because a driver still has some authority in avoiding obstacles and because the velocity of the vehicle is quickly diminishing, often leaving the vehicle stopped on the side of the road. Naturally, the best case outcome is neither LOC nor RD. An automated algorithm to detect LOC and RD was developed. The algorithm first detects if the drive was ended prematurely by the simulator, in which case the terminal yaw rate and yaw acceleration are used to determine if LOC would have been inevitable. If the maneuver was completed, the slip rate is used as a first-level determination. If the slip rate never exceeded a threshold and the vehicle came to a complete stop, the actual yaw of the vehicle relative to the road is used to determine LOC. The detailed implementation of this algorithm is beyond the scope of this paper; however, details can be found in another paper [2]. For this study, the algorithm was coded in software and applied to all participants. To ensure that the algorithm operated in a way that still maintained some connection to a subjective Watson, Papelis and Ahmad 10 judgment of LOC, four human raters reviewed study video to determine LOC. A simplified set of criteria was used as guidance to the human raters. A preliminary comparison yielded a set of consistent differences between the raters and the automatic algorithm. Careful inspection indicated that a software bug had caused the automatic algorithm to misjudge these maneuvers. After correcting this software error, the algorithm was reapplied and the final data were compared to assess inter-rater reliability as well as human vs. automatic evaluation. Each scenario presented additional challenges, in particular in maintaining consistency of exposure across the large number of participants. The primary concern for the curve scenario was maintaining a consistent entry speed. Figure 8 shows the speed of all participants at the three points in the curve. The first point (Crv_EntrSpd1) marks the location 400 ft before the first speed limit sign (50 mph) before the first right curve. The second point (Crv_EntrSpd2) marks the location of the second speed limit sign (45 mph) after the first right curve and before the left curve. The third point (Crv_EntrSped3) marks the location right before the start of the sharp left curve. Average speeds were 63.8 mph, 45.2 mph and 43.3 mph for the three points. The standard deviations are 1.4, 5.4, and 5.2 mph, respectively. As evident by these results, road signage and the briefing instructions given to the participants were successful in ensuring consistent exposure for all drivers in this scenario. Curve Scenario Entry Speeds 0 10 20 30 40 50 60 70 80 Exposure m ph Crv_EntrSpd1